{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.9.0\n",
      "\n",
      "torch 1.7.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Runs on CPU or GPU (if available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Zoo -- Reproducible Results with Deterministic Behavior and Runtime Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we are benchmarking the performance impact of setting PyTorch to deterministic behavior. In general, there are two aspects for reproducible resuls in PyTorch, \n",
    "1. Setting a random seed\n",
    "2. Setting cuDNN and PyTorch algorithmic behavior to deterministic\n",
    "\n",
    "For more details, please see https://pytorch.org/docs/stable/notes/randomness.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Setting a random seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I recommend using a function like the following one prior to using dataset loaders and initializing a model if you want to ensure the data is shuffled in the same manner if you rerun this notebook and the model gets the same initial random weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_all_seeds(seed):\n",
    "    os.environ[\"PL_GLOBAL_SEED\"] = str(seed)\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    torch.cuda.manual_seed_all(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Setting cuDNN and PyTorch algorithmic behavior to deterministic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the `set_all_seeds` function above, I recommend setting the behavior of PyTorch and cuDNN to deterministic (this is particulary relevant when using GPUs). We can also define a function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_deterministic():\n",
    "    if torch.cuda.is_available():\n",
    "        torch.backends.cudnn.benchmark = False\n",
    "        torch.backends.cudnn.deterministic = True\n",
    "    torch.set_deterministic(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1) Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After setting up the general configuration in this section, the following two sections will train a ResNet-101 model without and with deterministic behavior to get a sense how using deterministic options affect the runtime speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device: cuda:0\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Device\n",
    "DEVICE = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print('Device:', DEVICE)\n",
    "\n",
    "# Data settings\n",
    "num_classes = 10\n",
    "\n",
    "# Hyperparameters\n",
    "random_seed = 1\n",
    "learning_rate = 0.01\n",
    "batch_size = 128\n",
    "num_epochs = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2) Run without Deterministic Behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we enable deterministic behavior, we will run a ResNet-101 with otherwise the exact same settings for comparison. Note that setting random seeds doesn't affect the timing results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Set random seed ###\n",
    "set_all_seeds(random_seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### Dataset\n",
    "##########################\n",
    "\n",
    "from deterministic_benchmark_utils import get_dataloaders\n",
    "\n",
    "\n",
    "train_loader, test_loader = get_dataloaders(batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### Model\n",
    "##########################\n",
    "\n",
    "\n",
    "from deterministic_benchmark_utils import resnet101\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "model = resnet101(num_classes, grayscale=False)\n",
    "\n",
    "model = model.to(DEVICE)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/050 | Batch 0000/0391 | Loss: 2.8256\n",
      "Epoch: 001/050 | Batch 0200/0391 | Loss: 3.0516\n",
      "Epoch: 001/050 | Train: 25.898% |  Loss: 2.928\n",
      "Time elapsed: 1.27 min\n",
      "Epoch: 002/050 | Batch 0000/0391 | Loss: 1.8602\n",
      "Epoch: 002/050 | Batch 0200/0391 | Loss: 1.7631\n",
      "Epoch: 002/050 | Train: 18.342% |  Loss: 2.633\n",
      "Time elapsed: 2.60 min\n",
      "Epoch: 003/050 | Batch 0000/0391 | Loss: 1.5923\n",
      "Epoch: 003/050 | Batch 0200/0391 | Loss: 1.4562\n",
      "Epoch: 003/050 | Train: 43.756% |  Loss: 1.613\n",
      "Time elapsed: 3.92 min\n",
      "Epoch: 004/050 | Batch 0000/0391 | Loss: 1.3303\n",
      "Epoch: 004/050 | Batch 0200/0391 | Loss: 1.4450\n",
      "Epoch: 004/050 | Train: 55.988% |  Loss: 1.224\n",
      "Time elapsed: 5.23 min\n",
      "Epoch: 005/050 | Batch 0000/0391 | Loss: 1.2052\n",
      "Epoch: 005/050 | Batch 0200/0391 | Loss: 1.2398\n",
      "Epoch: 005/050 | Train: 59.410% |  Loss: 1.172\n",
      "Time elapsed: 6.55 min\n",
      "Epoch: 006/050 | Batch 0000/0391 | Loss: 1.2383\n",
      "Epoch: 006/050 | Batch 0200/0391 | Loss: 1.0209\n",
      "Epoch: 006/050 | Train: 64.646% |  Loss: 1.032\n",
      "Time elapsed: 7.88 min\n",
      "Epoch: 007/050 | Batch 0000/0391 | Loss: 0.9393\n",
      "Epoch: 007/050 | Batch 0200/0391 | Loss: 0.9948\n",
      "Epoch: 007/050 | Train: 64.638% |  Loss: 1.032\n",
      "Time elapsed: 9.22 min\n",
      "Epoch: 008/050 | Batch 0000/0391 | Loss: 0.9120\n",
      "Epoch: 008/050 | Batch 0200/0391 | Loss: 1.0325\n",
      "Epoch: 008/050 | Train: 32.180% |  Loss: 18.213\n",
      "Time elapsed: 10.54 min\n",
      "Epoch: 009/050 | Batch 0000/0391 | Loss: 1.1591\n",
      "Epoch: 009/050 | Batch 0200/0391 | Loss: 0.9179\n",
      "Epoch: 009/050 | Train: 69.116% |  Loss: 1.124\n",
      "Time elapsed: 11.87 min\n",
      "Epoch: 010/050 | Batch 0000/0391 | Loss: 0.8766\n",
      "Epoch: 010/050 | Batch 0200/0391 | Loss: 0.6505\n",
      "Epoch: 010/050 | Train: 71.980% |  Loss: 0.830\n",
      "Time elapsed: 13.21 min\n",
      "Epoch: 011/050 | Batch 0000/0391 | Loss: 0.5352\n",
      "Epoch: 011/050 | Batch 0200/0391 | Loss: 0.5256\n",
      "Epoch: 011/050 | Train: 68.338% |  Loss: 2.584\n",
      "Time elapsed: 14.53 min\n",
      "Epoch: 012/050 | Batch 0000/0391 | Loss: 0.6155\n",
      "Epoch: 012/050 | Batch 0200/0391 | Loss: 0.9926\n",
      "Epoch: 012/050 | Train: 76.710% |  Loss: 0.706\n",
      "Time elapsed: 15.84 min\n",
      "Epoch: 013/050 | Batch 0000/0391 | Loss: 0.6697\n",
      "Epoch: 013/050 | Batch 0200/0391 | Loss: 0.7854\n",
      "Epoch: 013/050 | Train: 63.266% |  Loss: 1.133\n",
      "Time elapsed: 17.16 min\n",
      "Epoch: 014/050 | Batch 0000/0391 | Loss: 0.6116\n",
      "Epoch: 014/050 | Batch 0200/0391 | Loss: 1.4822\n",
      "Epoch: 014/050 | Train: 49.220% |  Loss: 1.515\n",
      "Time elapsed: 18.47 min\n",
      "Epoch: 015/050 | Batch 0000/0391 | Loss: 1.0279\n",
      "Epoch: 015/050 | Batch 0200/0391 | Loss: 0.8855\n",
      "Epoch: 015/050 | Train: 74.914% |  Loss: 0.794\n",
      "Time elapsed: 19.78 min\n",
      "Epoch: 016/050 | Batch 0000/0391 | Loss: 0.7520\n",
      "Epoch: 016/050 | Batch 0200/0391 | Loss: 0.6673\n",
      "Epoch: 016/050 | Train: 10.000% |  Loss: 6543.246\n",
      "Time elapsed: 21.09 min\n",
      "Epoch: 017/050 | Batch 0000/0391 | Loss: 0.9921\n",
      "Epoch: 017/050 | Batch 0200/0391 | Loss: 0.6825\n",
      "Epoch: 017/050 | Train: 81.216% |  Loss: 0.560\n",
      "Time elapsed: 22.42 min\n",
      "Epoch: 018/050 | Batch 0000/0391 | Loss: 0.4659\n",
      "Epoch: 018/050 | Batch 0200/0391 | Loss: 0.7473\n",
      "Epoch: 018/050 | Train: 76.878% |  Loss: 0.664\n",
      "Time elapsed: 23.73 min\n",
      "Epoch: 019/050 | Batch 0000/0391 | Loss: 0.3604\n",
      "Epoch: 019/050 | Batch 0200/0391 | Loss: 0.7030\n",
      "Epoch: 019/050 | Train: 77.634% |  Loss: 0.650\n",
      "Time elapsed: 25.04 min\n",
      "Epoch: 020/050 | Batch 0000/0391 | Loss: 0.5016\n",
      "Epoch: 020/050 | Batch 0200/0391 | Loss: 0.8234\n",
      "Epoch: 020/050 | Train: 61.298% |  Loss: 1.171\n",
      "Time elapsed: 26.35 min\n",
      "Epoch: 021/050 | Batch 0000/0391 | Loss: 0.9311\n",
      "Epoch: 021/050 | Batch 0200/0391 | Loss: 0.8169\n",
      "Epoch: 021/050 | Train: 73.246% |  Loss: 1.120\n",
      "Time elapsed: 27.67 min\n",
      "Epoch: 022/050 | Batch 0000/0391 | Loss: 0.7056\n",
      "Epoch: 022/050 | Batch 0200/0391 | Loss: 0.4938\n",
      "Epoch: 022/050 | Train: 78.428% |  Loss: 0.695\n",
      "Time elapsed: 28.97 min\n",
      "Epoch: 023/050 | Batch 0000/0391 | Loss: 0.4753\n",
      "Epoch: 023/050 | Batch 0200/0391 | Loss: 0.2729\n",
      "Epoch: 023/050 | Train: 88.646% |  Loss: 0.335\n",
      "Time elapsed: 30.28 min\n",
      "Epoch: 024/050 | Batch 0000/0391 | Loss: 0.2956\n",
      "Epoch: 024/050 | Batch 0200/0391 | Loss: 0.5335\n",
      "Epoch: 024/050 | Train: 91.400% |  Loss: 0.256\n",
      "Time elapsed: 31.60 min\n",
      "Epoch: 025/050 | Batch 0000/0391 | Loss: 0.2332\n",
      "Epoch: 025/050 | Batch 0200/0391 | Loss: 0.2756\n",
      "Epoch: 025/050 | Train: 91.658% |  Loss: 0.246\n",
      "Time elapsed: 32.91 min\n",
      "Epoch: 026/050 | Batch 0000/0391 | Loss: 0.2198\n",
      "Epoch: 026/050 | Batch 0200/0391 | Loss: 0.2732\n",
      "Epoch: 026/050 | Train: 92.104% |  Loss: 0.234\n",
      "Time elapsed: 34.25 min\n",
      "Epoch: 027/050 | Batch 0000/0391 | Loss: 0.1831\n",
      "Epoch: 027/050 | Batch 0200/0391 | Loss: 0.1044\n",
      "Epoch: 027/050 | Train: 85.946% |  Loss: 0.423\n",
      "Time elapsed: 35.57 min\n",
      "Epoch: 028/050 | Batch 0000/0391 | Loss: 0.1531\n",
      "Epoch: 028/050 | Batch 0200/0391 | Loss: 0.1312\n",
      "Epoch: 028/050 | Train: 92.682% |  Loss: 0.217\n",
      "Time elapsed: 36.89 min\n",
      "Epoch: 029/050 | Batch 0000/0391 | Loss: 0.0809\n",
      "Epoch: 029/050 | Batch 0200/0391 | Loss: 0.1528\n",
      "Epoch: 029/050 | Train: 79.028% |  Loss: 0.754\n",
      "Time elapsed: 38.22 min\n",
      "Epoch: 030/050 | Batch 0000/0391 | Loss: 0.1996\n",
      "Epoch: 030/050 | Batch 0200/0391 | Loss: 0.1293\n",
      "Epoch: 030/050 | Train: 64.054% |  Loss: 1.079\n",
      "Time elapsed: 39.54 min\n",
      "Epoch: 031/050 | Batch 0000/0391 | Loss: 0.9535\n",
      "Epoch: 031/050 | Batch 0200/0391 | Loss: 0.8501\n",
      "Epoch: 031/050 | Train: 42.958% |  Loss: 1.706\n",
      "Time elapsed: 40.86 min\n",
      "Epoch: 032/050 | Batch 0000/0391 | Loss: 1.3670\n",
      "Epoch: 032/050 | Batch 0200/0391 | Loss: 1.2100\n",
      "Epoch: 032/050 | Train: 69.278% |  Loss: 0.960\n",
      "Time elapsed: 42.19 min\n",
      "Epoch: 033/050 | Batch 0000/0391 | Loss: 0.8023\n",
      "Epoch: 033/050 | Batch 0200/0391 | Loss: 1.0658\n",
      "Epoch: 033/050 | Train: 74.958% |  Loss: 0.724\n",
      "Time elapsed: 43.51 min\n",
      "Epoch: 034/050 | Batch 0000/0391 | Loss: 0.7112\n",
      "Epoch: 034/050 | Batch 0200/0391 | Loss: 0.4732\n",
      "Epoch: 034/050 | Train: 81.088% |  Loss: 0.581\n",
      "Time elapsed: 44.83 min\n",
      "Epoch: 035/050 | Batch 0000/0391 | Loss: 0.5735\n",
      "Epoch: 035/050 | Batch 0200/0391 | Loss: 0.3047\n",
      "Epoch: 035/050 | Train: 91.972% |  Loss: 0.619\n",
      "Time elapsed: 46.15 min\n",
      "Epoch: 036/050 | Batch 0000/0391 | Loss: 0.2496\n",
      "Epoch: 036/050 | Batch 0200/0391 | Loss: 0.2978\n",
      "Epoch: 036/050 | Train: 85.378% |  Loss: 1.753\n",
      "Time elapsed: 47.47 min\n",
      "Epoch: 037/050 | Batch 0000/0391 | Loss: 0.3852\n",
      "Epoch: 037/050 | Batch 0200/0391 | Loss: 0.3619\n",
      "Epoch: 037/050 | Train: 93.840% |  Loss: 4.329\n",
      "Time elapsed: 48.80 min\n",
      "Epoch: 038/050 | Batch 0000/0391 | Loss: 0.1941\n",
      "Epoch: 038/050 | Batch 0200/0391 | Loss: 0.7037\n",
      "Epoch: 038/050 | Train: 85.806% |  Loss: 0.990\n",
      "Time elapsed: 50.12 min\n",
      "Epoch: 039/050 | Batch 0000/0391 | Loss: 0.4254\n",
      "Epoch: 039/050 | Batch 0200/0391 | Loss: 0.3473\n",
      "Epoch: 039/050 | Train: 94.922% |  Loss: 0.786\n",
      "Time elapsed: 51.45 min\n",
      "Epoch: 040/050 | Batch 0000/0391 | Loss: 0.2045\n",
      "Epoch: 040/050 | Batch 0200/0391 | Loss: 0.2296\n",
      "Epoch: 040/050 | Train: 96.172% |  Loss: 0.826\n",
      "Time elapsed: 52.78 min\n",
      "Epoch: 041/050 | Batch 0000/0391 | Loss: 0.1372\n",
      "Epoch: 041/050 | Batch 0200/0391 | Loss: 0.0460\n",
      "Epoch: 041/050 | Train: 97.086% |  Loss: 0.332\n",
      "Time elapsed: 54.11 min\n",
      "Epoch: 042/050 | Batch 0000/0391 | Loss: 0.0638\n",
      "Epoch: 042/050 | Batch 0200/0391 | Loss: 0.4592\n",
      "Epoch: 042/050 | Train: 98.026% |  Loss: 0.064\n",
      "Time elapsed: 55.43 min\n",
      "Epoch: 043/050 | Batch 0000/0391 | Loss: 0.0578\n",
      "Epoch: 043/050 | Batch 0200/0391 | Loss: 0.0900\n",
      "Epoch: 043/050 | Train: 98.144% |  Loss: 0.056\n",
      "Time elapsed: 56.75 min\n",
      "Epoch: 044/050 | Batch 0000/0391 | Loss: 0.0333\n",
      "Epoch: 044/050 | Batch 0200/0391 | Loss: 0.0402\n",
      "Epoch: 044/050 | Train: 96.896% |  Loss: 0.097\n",
      "Time elapsed: 58.08 min\n",
      "Epoch: 045/050 | Batch 0000/0391 | Loss: 0.0325\n",
      "Epoch: 045/050 | Batch 0200/0391 | Loss: 0.0192\n",
      "Epoch: 045/050 | Train: 97.144% |  Loss: 0.084\n",
      "Time elapsed: 59.40 min\n",
      "Epoch: 046/050 | Batch 0000/0391 | Loss: 0.0604\n",
      "Epoch: 046/050 | Batch 0200/0391 | Loss: 0.0147\n",
      "Epoch: 046/050 | Train: 97.222% |  Loss: 0.082\n",
      "Time elapsed: 60.72 min\n",
      "Epoch: 047/050 | Batch 0000/0391 | Loss: 0.0241\n",
      "Epoch: 047/050 | Batch 0200/0391 | Loss: 0.0300\n",
      "Epoch: 047/050 | Train: 98.188% |  Loss: 0.054\n",
      "Time elapsed: 62.03 min\n",
      "Epoch: 048/050 | Batch 0000/0391 | Loss: 0.0113\n",
      "Epoch: 048/050 | Batch 0200/0391 | Loss: 0.0640\n",
      "Epoch: 048/050 | Train: 96.772% |  Loss: 0.100\n",
      "Time elapsed: 63.35 min\n",
      "Epoch: 049/050 | Batch 0000/0391 | Loss: 0.0550\n",
      "Epoch: 049/050 | Batch 0200/0391 | Loss: 0.0509\n",
      "Epoch: 049/050 | Train: 97.618% |  Loss: 0.072\n",
      "Time elapsed: 64.68 min\n",
      "Epoch: 050/050 | Batch 0000/0391 | Loss: 0.0656\n",
      "Epoch: 050/050 | Batch 0200/0391 | Loss: 0.1157\n",
      "Epoch: 050/050 | Train: 96.584% |  Loss: 0.100\n",
      "Time elapsed: 66.00 min\n",
      "Total Training Time: 66.00 min\n"
     ]
    }
   ],
   "source": [
    "from deterministic_benchmark_utils import train\n",
    "\n",
    "train(num_epochs, train_loader, model, optimizer, DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3) Run with Deterministic Behavior"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we set the deterministic behavior via the `set_deterministic()` function defined at the top of this notebook and compare how it affects the runtime speed of the ResNet-101 model. (Note that setting random seeds doesn't affect the timing results.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_deterministic()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Set random seed ###\n",
    "set_all_seeds(random_seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "##########################\n",
    "### Dataset\n",
    "##########################\n",
    "\n",
    "from deterministic_benchmark_utils import get_dataloaders\n",
    "\n",
    "\n",
    "train_loader, test_loader = get_dataloaders(batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### Model\n",
    "##########################\n",
    "\n",
    "\n",
    "from deterministic_benchmark_utils import resnet101\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "model = resnet101(num_classes, grayscale=False)\n",
    "\n",
    "model = model.to(DEVICE)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/050 | Batch 0000/0391 | Loss: 2.8256\n",
      "Epoch: 001/050 | Batch 0200/0391 | Loss: 2.4138\n",
      "Epoch: 001/050 | Train: 24.776% |  Loss: 1.998\n",
      "Time elapsed: 1.32 min\n",
      "Epoch: 002/050 | Batch 0000/0391 | Loss: 1.8754\n",
      "Epoch: 002/050 | Batch 0200/0391 | Loss: 1.7266\n",
      "Epoch: 002/050 | Train: 32.594% |  Loss: 1.825\n",
      "Time elapsed: 2.64 min\n",
      "Epoch: 003/050 | Batch 0000/0391 | Loss: 1.5575\n",
      "Epoch: 003/050 | Batch 0200/0391 | Loss: 1.4953\n",
      "Epoch: 003/050 | Train: 43.622% |  Loss: 1.650\n",
      "Time elapsed: 3.95 min\n",
      "Epoch: 004/050 | Batch 0000/0391 | Loss: 1.4275\n",
      "Epoch: 004/050 | Batch 0200/0391 | Loss: 1.5480\n",
      "Epoch: 004/050 | Train: 46.606% |  Loss: 1.561\n",
      "Time elapsed: 5.28 min\n",
      "Epoch: 005/050 | Batch 0000/0391 | Loss: 1.3144\n",
      "Epoch: 005/050 | Batch 0200/0391 | Loss: 1.8645\n",
      "Epoch: 005/050 | Train: 36.120% |  Loss: 1.690\n",
      "Time elapsed: 6.60 min\n",
      "Epoch: 006/050 | Batch 0000/0391 | Loss: 1.7935\n",
      "Epoch: 006/050 | Batch 0200/0391 | Loss: 1.5850\n",
      "Epoch: 006/050 | Train: 44.868% |  Loss: 1.489\n",
      "Time elapsed: 7.92 min\n",
      "Epoch: 007/050 | Batch 0000/0391 | Loss: 1.3980\n",
      "Epoch: 007/050 | Batch 0200/0391 | Loss: 1.3428\n",
      "Epoch: 007/050 | Train: 47.808% |  Loss: 1.484\n",
      "Time elapsed: 9.24 min\n",
      "Epoch: 008/050 | Batch 0000/0391 | Loss: 1.4614\n",
      "Epoch: 008/050 | Batch 0200/0391 | Loss: 1.3019\n",
      "Epoch: 008/050 | Train: 52.422% |  Loss: 1.314\n",
      "Time elapsed: 10.56 min\n",
      "Epoch: 009/050 | Batch 0000/0391 | Loss: 1.2225\n",
      "Epoch: 009/050 | Batch 0200/0391 | Loss: 1.3248\n",
      "Epoch: 009/050 | Train: 55.952% |  Loss: 1.207\n",
      "Time elapsed: 11.89 min\n",
      "Epoch: 010/050 | Batch 0000/0391 | Loss: 1.0489\n",
      "Epoch: 010/050 | Batch 0200/0391 | Loss: 0.9770\n",
      "Epoch: 010/050 | Train: 56.532% |  Loss: 1.236\n",
      "Time elapsed: 13.21 min\n",
      "Epoch: 011/050 | Batch 0000/0391 | Loss: 0.9592\n",
      "Epoch: 011/050 | Batch 0200/0391 | Loss: 0.8355\n",
      "Epoch: 011/050 | Train: 64.696% |  Loss: 1.043\n",
      "Time elapsed: 14.52 min\n",
      "Epoch: 012/050 | Batch 0000/0391 | Loss: 0.8492\n",
      "Epoch: 012/050 | Batch 0200/0391 | Loss: 0.8292\n",
      "Epoch: 012/050 | Train: 70.080% |  Loss: 0.841\n",
      "Time elapsed: 15.85 min\n",
      "Epoch: 013/050 | Batch 0000/0391 | Loss: 0.7789\n",
      "Epoch: 013/050 | Batch 0200/0391 | Loss: 0.9653\n",
      "Epoch: 013/050 | Train: 70.460% |  Loss: 0.826\n",
      "Time elapsed: 17.17 min\n",
      "Epoch: 014/050 | Batch 0000/0391 | Loss: 0.7094\n",
      "Epoch: 014/050 | Batch 0200/0391 | Loss: 0.7303\n",
      "Epoch: 014/050 | Train: 71.660% |  Loss: 0.784\n",
      "Time elapsed: 18.49 min\n",
      "Epoch: 015/050 | Batch 0000/0391 | Loss: 0.6977\n",
      "Epoch: 015/050 | Batch 0200/0391 | Loss: 0.6456\n",
      "Epoch: 015/050 | Train: 76.040% |  Loss: 0.679\n",
      "Time elapsed: 19.81 min\n",
      "Epoch: 016/050 | Batch 0000/0391 | Loss: 0.6240\n",
      "Epoch: 016/050 | Batch 0200/0391 | Loss: 0.6386\n",
      "Epoch: 016/050 | Train: 78.260% |  Loss: 0.609\n",
      "Time elapsed: 21.14 min\n",
      "Epoch: 017/050 | Batch 0000/0391 | Loss: 0.6116\n",
      "Epoch: 017/050 | Batch 0200/0391 | Loss: 0.5054\n",
      "Epoch: 017/050 | Train: 74.028% |  Loss: 0.749\n",
      "Time elapsed: 22.46 min\n",
      "Epoch: 018/050 | Batch 0000/0391 | Loss: 0.5558\n",
      "Epoch: 018/050 | Batch 0200/0391 | Loss: 0.8838\n",
      "Epoch: 018/050 | Train: 79.218% |  Loss: 0.594\n",
      "Time elapsed: 23.78 min\n",
      "Epoch: 019/050 | Batch 0000/0391 | Loss: 0.4119\n",
      "Epoch: 019/050 | Batch 0200/0391 | Loss: 0.6599\n",
      "Epoch: 019/050 | Train: 80.772% |  Loss: 0.553\n",
      "Time elapsed: 25.11 min\n",
      "Epoch: 020/050 | Batch 0000/0391 | Loss: 0.3910\n",
      "Epoch: 020/050 | Batch 0200/0391 | Loss: 0.4325\n",
      "Epoch: 020/050 | Train: 85.062% |  Loss: 0.427\n",
      "Time elapsed: 26.43 min\n",
      "Epoch: 021/050 | Batch 0000/0391 | Loss: 0.3287\n",
      "Epoch: 021/050 | Batch 0200/0391 | Loss: 0.4783\n",
      "Epoch: 021/050 | Train: 86.270% |  Loss: 0.397\n",
      "Time elapsed: 27.75 min\n",
      "Epoch: 022/050 | Batch 0000/0391 | Loss: 0.4166\n",
      "Epoch: 022/050 | Batch 0200/0391 | Loss: 0.3362\n",
      "Epoch: 022/050 | Train: 86.176% |  Loss: 0.396\n",
      "Time elapsed: 29.07 min\n",
      "Epoch: 023/050 | Batch 0000/0391 | Loss: 0.3534\n",
      "Epoch: 023/050 | Batch 0200/0391 | Loss: 0.2804\n",
      "Epoch: 023/050 | Train: 89.708% |  Loss: 0.293\n",
      "Time elapsed: 30.39 min\n",
      "Epoch: 024/050 | Batch 0000/0391 | Loss: 0.2640\n",
      "Epoch: 024/050 | Batch 0200/0391 | Loss: 0.2744\n",
      "Epoch: 024/050 | Train: 51.820% |  Loss: 1.357\n",
      "Time elapsed: 31.71 min\n",
      "Epoch: 025/050 | Batch 0000/0391 | Loss: 1.3786\n",
      "Epoch: 025/050 | Batch 0200/0391 | Loss: 0.8324\n",
      "Epoch: 025/050 | Train: 73.552% |  Loss: 0.756\n",
      "Time elapsed: 33.03 min\n",
      "Epoch: 026/050 | Batch 0000/0391 | Loss: 0.5476\n",
      "Epoch: 026/050 | Batch 0200/0391 | Loss: 0.6150\n",
      "Epoch: 026/050 | Train: 82.164% |  Loss: 0.519\n",
      "Time elapsed: 34.35 min\n",
      "Epoch: 027/050 | Batch 0000/0391 | Loss: 0.4786\n",
      "Epoch: 027/050 | Batch 0200/0391 | Loss: 0.3030\n",
      "Epoch: 027/050 | Train: 86.016% |  Loss: 0.402\n",
      "Time elapsed: 35.66 min\n",
      "Epoch: 028/050 | Batch 0000/0391 | Loss: 0.1926\n",
      "Epoch: 028/050 | Batch 0200/0391 | Loss: 0.5277\n",
      "Epoch: 028/050 | Train: 91.904% |  Loss: 0.238\n",
      "Time elapsed: 36.98 min\n",
      "Epoch: 029/050 | Batch 0000/0391 | Loss: 0.2020\n",
      "Epoch: 029/050 | Batch 0200/0391 | Loss: 0.1529\n",
      "Epoch: 029/050 | Train: 85.632% |  Loss: 0.411\n",
      "Time elapsed: 38.29 min\n",
      "Epoch: 030/050 | Batch 0000/0391 | Loss: 0.4071\n",
      "Epoch: 030/050 | Batch 0200/0391 | Loss: 0.2003\n",
      "Epoch: 030/050 | Train: 51.078% |  Loss: 2.223\n",
      "Time elapsed: 39.61 min\n",
      "Epoch: 031/050 | Batch 0000/0391 | Loss: 1.0328\n",
      "Epoch: 031/050 | Batch 0200/0391 | Loss: 0.2598\n",
      "Epoch: 031/050 | Train: 90.702% |  Loss: 0.276\n",
      "Time elapsed: 40.93 min\n",
      "Epoch: 032/050 | Batch 0000/0391 | Loss: 0.0750\n",
      "Epoch: 032/050 | Batch 0200/0391 | Loss: 0.1826\n",
      "Epoch: 032/050 | Train: 93.550% |  Loss: 0.183\n",
      "Time elapsed: 42.25 min\n",
      "Epoch: 033/050 | Batch 0000/0391 | Loss: 0.1842\n",
      "Epoch: 033/050 | Batch 0200/0391 | Loss: 0.1306\n",
      "Epoch: 033/050 | Train: 93.782% |  Loss: 0.179\n",
      "Time elapsed: 43.57 min\n",
      "Epoch: 034/050 | Batch 0000/0391 | Loss: 0.1161\n",
      "Epoch: 034/050 | Batch 0200/0391 | Loss: 0.0806\n",
      "Epoch: 034/050 | Train: 95.210% |  Loss: 0.138\n",
      "Time elapsed: 44.88 min\n",
      "Epoch: 035/050 | Batch 0000/0391 | Loss: 0.0715\n",
      "Epoch: 035/050 | Batch 0200/0391 | Loss: 0.0957\n",
      "Epoch: 035/050 | Train: 92.866% |  Loss: 0.212\n",
      "Time elapsed: 46.20 min\n",
      "Epoch: 036/050 | Batch 0000/0391 | Loss: 0.0629\n",
      "Epoch: 036/050 | Batch 0200/0391 | Loss: 0.1140\n",
      "Epoch: 036/050 | Train: 91.610% |  Loss: 0.261\n",
      "Time elapsed: 47.51 min\n",
      "Epoch: 037/050 | Batch 0000/0391 | Loss: 0.0436\n",
      "Epoch: 037/050 | Batch 0200/0391 | Loss: 0.1732\n",
      "Epoch: 037/050 | Train: 94.644% |  Loss: 0.154\n",
      "Time elapsed: 48.83 min\n",
      "Epoch: 038/050 | Batch 0000/0391 | Loss: 0.0935\n",
      "Epoch: 038/050 | Batch 0200/0391 | Loss: 0.0482\n",
      "Epoch: 038/050 | Train: 97.404% |  Loss: 0.075\n",
      "Time elapsed: 50.15 min\n",
      "Epoch: 039/050 | Batch 0000/0391 | Loss: 0.1096\n",
      "Epoch: 039/050 | Batch 0200/0391 | Loss: 0.2429\n",
      "Epoch: 039/050 | Train: 95.960% |  Loss: 0.119\n",
      "Time elapsed: 51.47 min\n",
      "Epoch: 040/050 | Batch 0000/0391 | Loss: 0.0648\n",
      "Epoch: 040/050 | Batch 0200/0391 | Loss: 0.1792\n",
      "Epoch: 040/050 | Train: 95.984% |  Loss: 0.116\n",
      "Time elapsed: 52.78 min\n",
      "Epoch: 041/050 | Batch 0000/0391 | Loss: 0.0663\n",
      "Epoch: 041/050 | Batch 0200/0391 | Loss: 0.0373\n",
      "Epoch: 041/050 | Train: 98.002% |  Loss: 0.059\n",
      "Time elapsed: 54.09 min\n",
      "Epoch: 042/050 | Batch 0000/0391 | Loss: 0.0611\n",
      "Epoch: 042/050 | Batch 0200/0391 | Loss: 0.0773\n",
      "Epoch: 042/050 | Train: 97.200% |  Loss: 0.080\n",
      "Time elapsed: 55.41 min\n",
      "Epoch: 043/050 | Batch 0000/0391 | Loss: 0.0216\n",
      "Epoch: 043/050 | Batch 0200/0391 | Loss: 0.0328\n",
      "Epoch: 043/050 | Train: 95.320% |  Loss: 0.140\n",
      "Time elapsed: 56.72 min\n",
      "Epoch: 044/050 | Batch 0000/0391 | Loss: 0.0737\n",
      "Epoch: 044/050 | Batch 0200/0391 | Loss: 0.0514\n",
      "Epoch: 044/050 | Train: 97.866% |  Loss: 0.062\n",
      "Time elapsed: 58.04 min\n",
      "Epoch: 045/050 | Batch 0000/0391 | Loss: 0.0675\n",
      "Epoch: 045/050 | Batch 0200/0391 | Loss: 0.0465\n",
      "Epoch: 045/050 | Train: 72.038% |  Loss: 1.062\n",
      "Time elapsed: 59.35 min\n",
      "Epoch: 046/050 | Batch 0000/0391 | Loss: 0.9005\n",
      "Epoch: 046/050 | Batch 0200/0391 | Loss: 1.3599\n",
      "Epoch: 046/050 | Train: 75.970% |  Loss: 0.743\n",
      "Time elapsed: 60.66 min\n",
      "Epoch: 047/050 | Batch 0000/0391 | Loss: 0.6000\n",
      "Epoch: 047/050 | Batch 0200/0391 | Loss: 0.4327\n",
      "Epoch: 047/050 | Train: 91.000% |  Loss: 0.429\n",
      "Time elapsed: 61.98 min\n",
      "Epoch: 048/050 | Batch 0000/0391 | Loss: 0.3004\n",
      "Epoch: 048/050 | Batch 0200/0391 | Loss: 0.3143\n",
      "Epoch: 048/050 | Train: 96.802% |  Loss: 0.215\n",
      "Time elapsed: 63.29 min\n",
      "Epoch: 049/050 | Batch 0000/0391 | Loss: 0.1031\n",
      "Epoch: 049/050 | Batch 0200/0391 | Loss: 0.0739\n",
      "Epoch: 049/050 | Train: 97.926% |  Loss: 0.072\n",
      "Time elapsed: 64.60 min\n",
      "Epoch: 050/050 | Batch 0000/0391 | Loss: 0.0717\n",
      "Epoch: 050/050 | Batch 0200/0391 | Loss: 0.1018\n",
      "Epoch: 050/050 | Train: 96.278% |  Loss: 0.108\n",
      "Time elapsed: 65.92 min\n",
      "Total Training Time: 65.92 min\n"
     ]
    }
   ],
   "source": [
    "from deterministic_benchmark_utils import train\n",
    "\n",
    "train(num_epochs, train_loader, model, optimizer, DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4) Result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular case, 66 min vs 65.92, the deterministic behavior does not seem to influence performance."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
